Analysing Errors of Open Information Extraction Systems

We report results on benchmarking Open Information Extraction (OIE) systems using RelVis, a toolkit for benchmarking Open Information Extraction systems. Our comprehensive benchmark contains three data sets from the news domain and one data set from Wikipedia with overall 4522 labeled sentences and 11243 binary or n-ary OIE relations. In our analysis on these data sets we compared the performance of four popular OIE systems, ClausIE, OpenIE 4.2, Stanford OpenIE and PredPatt. In addition, we evaluated the impact of five common error classes on a subset of 749 n-ary tuples. From our deep analysis we unreveal important research directions for a next generation of OIE systems.Comment: Accepted at Building Linguistically Generalizable NLP Systems at EMNLP 201

Is language modeling enough? Evaluating effective embedding combinations

Universal embeddings, such as BERT or ELMo, are useful for a broad set of natural language processing tasks like text classification or sentiment analysis. Moreover, specialized embeddings also exist for tasks like topic modeling or named entity disambiguation. We study if we can complement these universal embeddings with specialized embeddings. We conduct an in-depth evaluation of nine well known natural language understanding tasks with SentEval. Also, we extend SentEval with two additional tasks to the medical domain. We present PubMedSection, a novel topic classification dataset focussed on the biomedical domain. Our comprehensive analysis covers 11 tasks and combinations of six embeddings. We report that combined embeddings outperform state of the art universal embeddings without any embedding fine-tuning. We observe that adding topic model based embeddings helps for most tasks and that differing pre-training tasks encode complementary features. Moreover, we present new state of the art results on the MPQA and SUBJ tasks in SentEval

tsnlp Database

this report. To allow for the dissemination and wide distribution of tsnlp results, obviously the technology developed has to be of almost industrial strength. Therefore, the dabase design primarily puts emphasis on engineering rather than on scientific aspects such that simplicity, robustness and documentation of the software play a very important role. Because the tools implemented in work package 6 will have to be documented to an extent that they can be used by outside parties, it is foreseen that a revised version of the present report will be made available after the completion of the implementation. The tsnlp Database --- From tsct to tsdb 2 The Test Suite Construction Tool (tsct

The TSNLP Database - From tsct to tsdb

this report. To allow for the dissemination and wide distribution of tsnlp results, obviously the technology developed has to be of almost industrial strength. Therefore, the dabase design primarily puts emphasis on engineering rather than on scientific aspects such that simplicity, robustness and documentation of the software play a very important role. Because the tools implemented in work package 6 will have to be documented to an extent that they can be used by outside parties, it is foreseen that a revised version of the present report will be made available after the completion of the implementation. The tsnlp Database --- From tsct to tsdb 2 The Test Suite Construction Tool (tsct

TSNLP User Manual. Volume 2: Core Test Suite Technology

Evangelia Kordoni, Siety Meijer, and Martin Rondell (University of Essex) and Judith Baur, Tom Fettig, Judith Klein, and Fred Oberhauser (DFKI). The User Manual is in three volumes: ffl Volume 1: Background, Methodology, Customisation and Testing Lorna Balkan, Frederik Fouvry, Sylvie Regnier-Prost (editors); ffl Volume 2: Core Test Suite Technology Stephan Oepen, Frederik Fouvry, Klaus Netter, Tom Fettig, Fred Oberhauser; ffl Volume 2b: Test Suite Technology Frederik Fouvry (editor); 1 Full postal addresses and phone numbers for contact persons of the consortium members are: CLjMT Group ISSCO Aerospatiale France DFKI GmbH University of Essex Universit'e de Gen`eve Common Research Center CL Department Wivenhoe Park 54, route des Acacias 12, rue Pasteur BP 76 Stuhlsatzenhausweg 3 UK Colchester CO4 3SQ CH 1227 Gen`eve F 92152 Suresnes Cedex D 66123 Saarbrucken +44 - 1206 - 87 20 86 +41 - 22 - 705 787

Incidence of severe critical events in paediatric anaesthesia (APRICOT): a prospective multicentre observational study in 261 hospitals in Europe

Background Little is known about the incidence of severe critical events in children undergoing general anaesthesia in Europe. We aimed to identify the incidence, nature, and outcome of severe critical events in children undergoing anaesthesia, and the associated potential risk factors. Methods The APRICOT study was a prospective observational multicentre cohort study of children from birth to 15 years of age undergoing elective or urgent anaesthesia for diagnostic or surgical procedures. Children were eligible for inclusion during a 2-week period determined prospectively by each centre. There were 261 participating centres across 33 European countries. The primary endpoint was the occurence of perioperative severe critical events requiring immediate intervention. A severe critical event was defined as the occurrence of respiratory, cardiac, allergic, or neurological complications requiring immediate intervention and that led (or could have led) to major disability or death. This study is registered with ClinicalTrials.gov, number NCT01878760. Findings Between April 1, 2014, and Jan 31, 2015, 31â127 anaesthetic procedures in 30â874 children with a mean age of 6·35 years (SD 4·50) were included. The incidence of perioperative severe critical events was 5·2% (95% CI 5·0â5·5) with an incidence of respiratory critical events of 3·1% (2·9â3·3). Cardiovascular instability occurred in 1·9% (1·7â2·1), with an immediate poor outcome in 5·4% (3·7â7·5) of these cases. The all-cause 30-day in-hospital mortality rate was 10 in 10â000. This was independent of type of anaesthesia. Age (relative risk 0·88, 95% CI 0·86â0·90; p<0·0001), medical history, and physical condition (1·60, 1·40â1·82; p<0·0001) were the major risk factors for a serious critical event. Multivariate analysis revealed evidence for the beneficial effect of years of experience of the most senior anaesthesia team member (0·99, 0·981â0·997; p<0·0048 for respiratory critical events, and 0·98, 0·97â0·99; p=0·0039 for cardiovascular critical events), rather than the type of health institution or providers. Interpretation This study highlights a relatively high rate of severe critical events during the anaesthesia management of children for surgical or diagnostic procedures in Europe, and a large variability in the practice of paediatric anaesthesia. These findings are substantial enough to warrant attention from national, regional, and specialist societies to target education of anaesthesiologists and their teams and implement strategies for quality improvement in paediatric anaesthesia. Funding European Society of Anaesthesiology

Incidence of severe critical events in paediatric anaesthesia (APRICOT): a prospective multicentre observational study in 261 hospitals in Europe

Background Little is known about the incidence of severe critical events in children undergoing general anaesthesia in Europe. We aimed to identify the incidence, nature, and outcome of severe critical events in children undergoing anaesthesia, and the associated potential risk factors. Methods The APRICOT study was a prospective observational multicentre cohort study of children from birth to 15 years of age undergoing elective or urgent anaesthesia for diagnostic or surgical procedures. Children were eligible for inclusion during a 2-week period determined prospectively by each centre. There were 261 participating centres across 33 European countries. The primary endpoint was the occurence of perioperative severe critical events requiring immediate intervention. A severe critical event was defined as the occurrence of respiratory, cardiac, allergic, or neurological complications requiring immediate intervention and that led (or could have led) to major disability or death. This study is registered with ClinicalTrials.gov, number NCT01878760. Findings Between April 1, 2014, and Jan 31, 2015, 31 127 anaesthetic procedures in 30 874 children with a mean age of 6.35 years (SD 4.50) were included. The incidence of perioperative severe critical events was 5.2% (95% CI 5.0-5.5) with an incidence of respiratory critical events of 3.1% (2.9-3.3). Cardiovascular instability occurred in 1.9% (1.7-2.1), with an immediate poor outcome in 5.4% (3.7-7.5) of these cases. The all-cause 30-day in-hospital mortality rate was 10 in 10 000. This was independent of type of anaesthesia. Age (relative risk 0.88, 95% CI 0.86-0.90; p<0.0001), medical history, and physical condition (1.60, 1.40-1.82; p<0.0001) were the major risk factors for a serious critical event. Multivariate analysis revealed evidence for the beneficial effect of years of experience of the most senior anaesthesia team member (0.99, 0.981-0.997; p<0.0048 for respiratory critical events, and 0.98, 0.97-0.99; p=0.0039 for cardiovascular critical events), rather than the type of health institution or providers. Interpretation This study highlights a relatively high rate of severe critical events during the anaesthesia management of children for surgical or diagnostic procedures in Europe, and a large variability in the practice of paediatric anaesthesia. These findings are substantial enough to warrant attention from national, regional, and specialist societies to target education of anaesthesiologists and their teams and implement strategies for quality improvement in paediatric anaesthesia